{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning Tutorial with Residual Nets (aka ResNets)  in PyTorch! <br> No equations, just code and comments!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are - similarly to me - fairly new to the field of Machine Learning (ML) in general, and Deep Learning in particular, then the enormous landscape of neural network architectures can often be pretty overwhelming. I guess, all newbies know what I am talking about. CNNs, RNNs, GRUs, LSTMs, U-Nets, Dense Nets, Residual Nets, Highway Nets, Transformers, the entire sesame street, etc.\n",
    "\n",
    "\n",
    "- Don't worry! We will not cover all of them in this little `PyTorch` tutorial. \n",
    "- Moreover, we will not use any equations! We will just use simple Python and PyTorch code.\n",
    "\n",
    "\n",
    "### Why ResNets?\n",
    "\n",
    "We will cover ResNets only. Why? \n",
    "\n",
    "- Firstly, because I believe that understanding `ResNets` will not only help to demystify the magic of neural networks, it will also lead to a more visual understanding of linear transformations and matrix-matrix operations in high-dimensional space. You will see that every architecture that appears complex at first sight is nothing more than applied linear algebra.\n",
    "    \n",
    "    \n",
    "- Secondly, it does require to be familiar with `Convolutional Neural Networks (CNNs)`. Again, don't worry! I don't expect you to be an expert in developing CNNs. If you have an intuitive understanding of convolutions and know how they work (e.g., convoluting a Gaussian kernel ($3x3$, $5x5$, etc.) with an equally sized window in some image), you will grasp the nature of CNNs along the way. If you have never hear of convolutions before, please have a look at them first before you move on. There is no point in skipping the small but indispensable bits. As ever, start with wikipedia: [Convolutions](https://en.wikipedia.org/wiki/Convolution)\n",
    "\n",
    "### Let's demystify ResNets and break it down!\n",
    "\n",
    "The first time, I've heard about Residual Nets (aka ResNets) I was thinking that this is something I will maybe, if I'm lucky, understand at some point in the distant future. \"ResNets must be beyond my understanding of maths and programming\" - those were my thoughts. Then, however, I've immersed myself in Convolutional Neural Networks and their big brother - Residual Networks. To be frank, ResNets are actually not that complex. Everyone who knows about Linear Algebra and has an understanding about the nature of linear transformations with vectors and matrices, will quickly grasp the mechanisms of ResNets. Let's dive right into it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install PyTorch\n",
    "\n",
    "If you haven't installed `PyTorch` on your machine, then install the latest version of PyTorch with the usual pip command <br> `pip install torch`.\n",
    "\n",
    "### Download MNIST\n",
    "\n",
    "Download the famous MNIST data set for Deep Learning from [here](http://deeplearning.net/data/mnist/). Create a `data` subfolder and store the data set there. You don't have to unzip it. We will do this automatically via Python.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.193040Z",
     "start_time": "2019-09-10T15:10:51.750594Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "import gzip\n",
    "import torch \n",
    "import numpy as np\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from torch import optim\n",
    "from torch.utils.data import TensorDataset\n",
    "from torch.utils.data import DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.199660Z",
     "start_time": "2019-09-10T15:10:54.193040Z"
    }
   },
   "outputs": [],
   "source": [
    "PATH = './data/mnist/'\n",
    "FILENAME = 'mnist.pkl.gz'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a little function that will both unzip the data and load it into memory.\n",
    "You might need to install `gzip` and `pickle` first before you can execute the cell below. Use the usual `pip` command.\n",
    "\n",
    "Since the MNIST data is stored as a `pickle` file, we have to open it in both `read` and `binary` mode. Hence, instead of the usual `open` command, we will `gzip.open` the data (which automatically unzips the data set for us). The `r` stands for read mode and the `b` stands for binary mode. You'll be reading a binary file. The data is already split into `train` and `dev` set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.215702Z",
     "start_time": "2019-09-10T15:10:54.203670Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_files(path,filename):\n",
    "    with gzip.open((PATH + FILENAME), \"rb\") as file:\n",
    "        ((x_train, y_train), (x_val, y_val), _) = pickle.load(file, encoding='latin-1')\n",
    "    return x_train, y_train, x_val, y_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-05T07:44:14.700177Z",
     "start_time": "2019-09-05T07:44:14.696166Z"
    }
   },
   "source": [
    "**Tensors**: Since we are developing our neural network in PyTorch, we have to deal with `tensors` instead of `arrays`. However, mathematically speaking there is no difference between tensors and arrays. Arrays such as tensors can be vectors or matrices. The difference lies in the speed of executions. Tensors can be casted onto the `GPU`, whereas `NumPy` array operations are executed on the `CPU`. Hence, tensor operations are faster and more convenient for Deep Learning, where we have to deal with multiple matrix-matrix operations at the same time and need the GPU to execute code rapidly.\n",
    "\n",
    "To convert the data into `tensors`, we will use Python's handy built-in `map` object that applies a particular function (in our case `torch.tensor`) on all elements passed to the second argumet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.232747Z",
     "start_time": "2019-09-10T15:10:54.220717Z"
    }
   },
   "outputs": [],
   "source": [
    "def tensor_map(x_train,y_train,x_val,y_val): return map(torch.tensor,(x_train,y_train,x_val,y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned at the beginning of the tutorial, `ResNets` are an advancement of `CNNs` and thus just a slightly more complicated version of Convolutional Neural Networks. \n",
    "\n",
    "Convolutional `2D` layers expect the data to be passed in `4D`: `Batch size, Channels (1 for grey-scale, 3 for RGB), Height, Width`. Therefore, we will write a little `preprocessing` function that can be applied to our mini batch input data before performing the convolution. In our case, `channel = 1` as MNIST images are grey-scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.241771Z",
     "start_time": "2019-09-10T15:10:54.234753Z"
    }
   },
   "outputs": [],
   "source": [
    "def preprocess(x):\n",
    "    return x.view(-1, 1, 28, 28)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Refactor as much as possible!**\n",
    "\n",
    "In deep ResNets we need many convolutional layers. That's why we will write a `conv` function for an `nn.Conv2d` layer to not call the latter multiple times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.256813Z",
     "start_time": "2019-09-10T15:10:54.245782Z"
    }
   },
   "outputs": [],
   "source": [
    "def conv(in_size, out_size, pad=1): \n",
    "    return nn.Conv2d(in_size, out_size, kernel_size=3, stride=2, padding=pad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deep deep in neural (or rather vector) space! Anyways, it's getting exciting!**\n",
    "\n",
    "The code cell below is maybe the most important part in this tutorial on `ResNets`. \n",
    "\n",
    "**Inheriting classes from PyTorch**\n",
    "\n",
    "To develop your own neural networks in `PyTorch`, you have to write your own Python `classes`. In Python, `classes` are Python's approach toward Object Oriented Programming (aka OOP). We have to inherit from PyTorch's `nn.Module` class which is used for all neural network architectures in PyTorch and includes automatic `backpropagation` through time among other neat features. With Python's `super()` we can use the `init` method from the caring `parent` nn.Module that provides us with all its features without having to implement them ourselves. One can think of this as a newborn child called `ResBlock` inheriting and using all the `methods` from the already developed parent called `nn.Module`. \n",
    "\n",
    "\n",
    "**Residual Block**\n",
    "\n",
    "A `ResBlock` consists of a `ConvBlock` plus a `skip connection`. \n",
    "\n",
    "**ConvBlock**: Two times a convolution `conv` (look at the conv function in the cell above), followed by a `batch norm` layer. If you don't know what a batch normalization does and why it is quite useful in neural networks have a look [here](https://en.wikipedia.org/wiki/Batch_normalization). Spoiler: It makes neural networks more robust and stable and has proven to lead to better results. The latter is the reason why we apply a `batch norm` layer on top of each `conv` layer. Last but not least, we apply a `non-linearity`, here a rectified linear unit (aka `relu`), which is already implemented for us in PyTorch's `functional` module.\n",
    "\n",
    "\n",
    "**Skip connection**: After each conv block, we will add the input that was not passed through the convolutional block to the output of the conv block: `x + self.convblock(x)`. Like that, we keep information from the input that might get lost through the convolutions and applied non-linearities, and add them to the convoluted output. It's called `skip connection` because we skip the conv block and add it to the output of the conv block (basically all convolutions and non-linearities are skipped and we keep information from previous points in space and time). Experiments have shown that this leads to more robust and accurate networks.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.269846Z",
     "start_time": "2019-09-10T15:10:54.259819Z"
    }
   },
   "outputs": [],
   "source": [
    "class ResBlock(nn.Module):\n",
    "    \n",
    "    def __init__(self, in_size:int, hidden_size:int, out_size:int, pad:int):\n",
    "        super().__init__()\n",
    "        self.conv1 = conv(in_size, hidden_size, pad)\n",
    "        self.conv2 = conv(hidden_size, out_size, pad)\n",
    "        self.batchnorm1 = nn.BatchNorm2d(hidden_size)\n",
    "        self.batchnorm2 = nn.BatchNorm2d(out_size)\n",
    "    \n",
    "    def convblock(self, x):\n",
    "        x = F.relu(self.batchnorm1(self.conv1(x)))\n",
    "        x = F.relu(self.batchnorm2(self.conv2(x)))\n",
    "        return x\n",
    "    \n",
    "    def forward(self, x): return x + self.convblock(x) # skip connection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ResNet** \n",
    "\n",
    "Here we are creating our ResNet that applies the ConvBlock and the ResBlock which we have already implemented in the cells above. At the end we perform `max-pooling` but make it adaptive instead of fixed. Adpative max-poolig is a neat trick that allows us to pass inputs of any size (we only need to specify the number of outputs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.284887Z",
     "start_time": "2019-09-10T15:10:54.273858Z"
    }
   },
   "outputs": [],
   "source": [
    "class ResNet(nn.Module):\n",
    "    \n",
    "    def __init__(self, n_classes=10):\n",
    "        super().__init__()\n",
    "        self.res1 = ResBlock(1, 8, 16, 15)\n",
    "        self.res2 = ResBlock(16, 32, 16, 15)\n",
    "        self.conv = conv(16, n_classes)\n",
    "        self.batchnorm = nn.BatchNorm2d(n_classes)\n",
    "        self.maxpool = nn.AdaptiveMaxPool2d(1)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = preprocess(x)\n",
    "        x = self.res1(x)\n",
    "        x = self.res2(x) \n",
    "        x = self.maxpool(self.batchnorm(self.conv(x)))\n",
    "        return x.view(x.size(0), -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loss computation and batches**\n",
    "\n",
    "We will write a handy function that takes in our `ResNet`, a specified `loss function`, `mini batches` (input and target data), an optional `optimizer` (only needed for training but not for evaluation mode) and an optional `learning rate scheduler`. \n",
    "\n",
    "Thanks to PyTorch´s automatic gradient computation we don't have to write backprop manually (that would indeed be a hustle) and can just tell our optimizer to perform a step according to our `learning rate` and zero the gradients after each iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.296917Z",
     "start_time": "2019-09-10T15:10:54.287894Z"
    }
   },
   "outputs": [],
   "source": [
    "def loss_batch(model, loss_func, xb, yb, opt=None, scheduler=None):\n",
    "    loss = loss_func(model(xb), yb)\n",
    "    acc = accuracy(model(xb), yb)\n",
    "    if opt is not None:\n",
    "        loss.backward()\n",
    "        if scheduler is not None:\n",
    "            scheduler.step()\n",
    "        opt.step()\n",
    "        opt.zero_grad()\n",
    "    return acc, loss.item(), len(xb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.321986Z",
     "start_time": "2019-09-10T15:10:54.313967Z"
    }
   },
   "outputs": [],
   "source": [
    "def accuracy(out, yb):\n",
    "    # in PyTorch one cannot take the mean of ints \n",
    "    # thus, values have to be converted into floats first\n",
    "    preds = torch.argmax(out, dim=1)\n",
    "    return (preds == yb).float().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's get the model!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.308962Z",
     "start_time": "2019-09-10T15:10:54.298923Z"
    },
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "def get_model():\n",
    "    model = ResNet()\n",
    "    optimizer = optim.SGD(model.parameters(), lr=lr, momentum=0.9)\n",
    "    return model, optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's get the data!**\n",
    "\n",
    "`DataLoader` is an amazing PyTorch feature that will automatically split our train and dev data sets into <br> `math.ceil(x_train.shape[0]/batch size)` mini batches according to a given size (e.g., 64, 128) and optionally shuffle the data (only necessary for trainig). That is important for training as PyTorch's `nn.Module` requires inputs to be passed to the neural network in the form of tensor mini batches. However, we first have to create train and val data sets of tensors with `TensorDataset`. We will do all of the above in a single function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.335019Z",
     "start_time": "2019-09-10T15:10:54.324993Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_data_batches(x_train, y_train, x_val, y_val, bs):\n",
    "    train_ds = TensorDataset(x_train, y_train)\n",
    "    val_ds = TensorDataset(x_val, y_val)\n",
    "    # DataLoader = generator\n",
    "    return (\n",
    "        DataLoader(train_ds, batch_size=bs, shuffle=True),\n",
    "        DataLoader(val_ds, batch_size=bs * 2),\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's fit our model!**\n",
    "\n",
    "If the model is in training mode `model.train()` gradient computation is performed automatically. In evaluation mode `model.eval()`, however, we don't want gradient computation. Thus, we have to tell PyTorch with `with torch.no_grad()` that backprop should not be performed when testing our model on the dev set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.349057Z",
     "start_time": "2019-09-10T15:10:54.339031Z"
    }
   },
   "outputs": [],
   "source": [
    "def fit(epochs, model, loss_func, opt, train_dl, valid_dl, scheduler=None):\n",
    "    for epoch in range(epochs):\n",
    "        model.train()\n",
    "        # iterate over data loader object (generator)\n",
    "        for xb, yb in train_dl:\n",
    "            loss_batch(model, loss_func, xb, yb, opt, scheduler)\n",
    "\n",
    "        model.eval()\n",
    "        # no gradient computation for evaluation mode\n",
    "        with torch.no_grad():\n",
    "            accs, losses, nums = zip(\n",
    "                *[loss_batch(model, loss_func, xb, yb) for xb, yb in valid_dl]\n",
    "            )\n",
    "        \n",
    "        #NOTE: important to multiply with batch size and sum over values \n",
    "        #      to account for varying batch sizes\n",
    "        val_loss = np.sum(np.multiply(losses, nums)) / np.sum(nums)\n",
    "        val_acc = np.sum(np.multiply(accs, nums)) / np.sum(nums)\n",
    "\n",
    "        print(\"Epoch:\", epoch+1)\n",
    "        print(\"Loss: \", val_loss)\n",
    "        print(\"Accuracy: \", val_acc)\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hyperparameters**\n",
    "\n",
    "Define batch size `br`, learning rate `lr`, number of epochs `n_epochs` and loss function `loss_func`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:54.359083Z",
     "start_time": "2019-09-10T15:10:54.352065Z"
    }
   },
   "outputs": [],
   "source": [
    "bs=64 #128\n",
    "lr=0.01\n",
    "n_epochs = 5\n",
    "loss_func = F.cross_entropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:56.414449Z",
     "start_time": "2019-09-10T15:10:54.361090Z"
    }
   },
   "outputs": [],
   "source": [
    "# get data set\n",
    "x_train, y_train, x_val, y_val = get_files(PATH, FILENAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:57.053866Z",
     "start_time": "2019-09-10T15:10:56.416379Z"
    }
   },
   "outputs": [],
   "source": [
    "# map tensor function to all inputs (X) and targets (Y) to create tensor data sets\n",
    "x_train, y_train, x_val, y_val = tensor_map(x_train, y_train, x_val, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:57.059883Z",
     "start_time": "2019-09-10T15:10:57.055872Z"
    }
   },
   "outputs": [],
   "source": [
    "# get math.ceil(x_train.shape[0]/batch size) train and val mini batches of size bs\n",
    "train_dl, val_dl = get_data_batches(x_train, y_train, x_val, y_val, bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:10:57.227330Z",
     "start_time": "2019-09-10T15:10:57.063895Z"
    }
   },
   "outputs": [],
   "source": [
    "# get model and optimizer\n",
    "model, opt = get_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-10T15:56:14.482644Z",
     "start_time": "2019-09-10T15:10:57.230337Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0\n",
      "Loss:  0.18226656237840652\n",
      "Accuracy:  0.944\n",
      "Epoch: 1\n",
      "Loss:  0.1558196894288063\n",
      "Accuracy:  0.953\n",
      "Epoch: 2\n",
      "Loss:  0.11653886579871178\n",
      "Accuracy:  0.9653\n",
      "Epoch: 3\n",
      "Loss:  0.10886165248155594\n",
      "Accuracy:  0.9664\n",
      "Epoch: 4\n",
      "Loss:  0.10072250712513924\n",
      "Accuracy:  0.9693\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "fit(n_epochs, model, loss_func, opt, train_dl, val_dl)"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
